Fluorescence lifetime imaging using deep learning

ABSTRACT

One embodiment provides an apparatus for fluorescence lifetime imaging (FLI). The apparatus includes a deep neural network (DNN). The DNN includes a first convolutional layer, a plurality of intermediate layers and an output layer. The first convolutional layer is configured to receive FLI input data. Each intermediate layer is configured to receive a respective intermediate input corresponding to an output of a respective prior layer. Each intermediate layer is further configured to provide a respective intermediate output related to the received respective intermediate input. The output layer is configured to provide estimated FLI output data corresponding to the received FLI input data. The DNN is trained using synthetic data.

CROSS REFERENCE TO RELATED APPLICATION(S)

This application claims the benefit of U.S. Provisional Application No.62/958,022, filed Jan. 7, 2020, U.S. Provisional Application No.63/001,947, filed Mar. 30, 2020, and U.S. Provisional Application No.63/134,536, filed Jan. 6, 2021, which are incorporated by reference asif disclosed herein in their entireties.

GOVERNMENT LICENSE RIGHTS

This invention was made with government support under grants EB019443,CA207725, CA237267, and CA250636 awarded by the National Institutes ofHealth (NIH). The government has certain rights in the invention.

FIELD

The present disclosure relates to fluorescence lifetime imaging, inparticular to, fluorescence lifetime imaging using deep learning.

BACKGROUND

Fluorescence lifetime imaging (FLI) may be utilized in biomedical and/ormolecular biology applications. FLI generally provides quantitativeinformation, has a relatively high sensitivity, and is able tosimultaneously image a plurality of biomarkers and/or biologicalprocesses at various spatio-temporal scales. For example, FLI mayprovide insight into a cellular microenvironment by noninvasivelyexamining intracellular parameters including, but not limited to,metabolic status, reactive oxygen species, and intracellular pH.Additionally or alternatively, FLI may be utilized to quantify Försterresonance energy transfer (FRET), i.e., to quantify protein-proteininteractions, biosensor activity, and ligand-receptor engagement invivo.

In FLI, an acquired temporal data set is post-processed in order toquantify lifetime or lifetime-based parameters. Post-processingtypically includes a model-based process where iterative optimizationmethods are employed to estimate one or more parameters of interest(e.g., mean lifetime, FRET efficiencies and/or population fractions).Accuracy of the model-based processes may depend on user-definedparameter settings. Such user-defined parameter settings are susceptibleto user-related bias and may thus be a source of error. The model-basedprocesses are generally computationally expensive and relatively slow somay typically not be performed in real time.

SUMMARY

In some embodiments, there is provided an apparatus for fluorescencelifetime imaging (FLI). The apparatus includes a deep neural network(DNN). The DNN includes a first convolutional layer, a plurality ofintermediate layers, and an output layer. The first convolutional layeris configured to receive FLI input data. Each intermediate layer isconfigured to receive a respective intermediate input corresponding toan output of a respective prior layer. Each intermediate layer isfurther configured to provide a respective intermediate output relatedto the received respective intermediate input. The output layer isconfigured to provide estimated FLI output data corresponding to thereceived FLI input data. The DNN is trained using synthetic data.

In some embodiments of the apparatus, the first convolutional layer is athree-dimensional (3D) convolutional layer. The plurality ofintermediate layers includes a 3D residual block, a reshape layer, atwo-dimensional (2D) convolutional layer, and at least one 2D residualblock. The output layer is a fully convolutional (FC) down-sample layer.

In some embodiments of the apparatus, the first convolutional layer is aseparable two-dimensional (2D) convolutional layer. The plurality ofintermediate layers includes a 2D exception block, a 2D convolutionallayer, and a 2D residual block. The output layer is a fullyconvolutional (FC) down-sample layer.

In some embodiments, the apparatus further includes a discriminatornetwork configured to compare estimated training output data withtraining synthetic output data during training. The DNN and thediscriminator network correspond to a generative adversarial network(GAN) during training.

In some embodiments of the apparatus, the DNN further includes a secondconvolutional layer configured to receive optical property data. Theplurality of intermediate layers includes a concatenate layer. Theestimated FLI output data is further related to the optical propertydata.

In some embodiments of the apparatus, the first convolutional layer is athree-dimensional (3D) convolutional layer. The plurality ofintermediate layers includes a second 3D convolutional layer, a reshapelayer, a separable two-dimensional (2D) convolutional layer, and a 2Dexception block. The DNN includes a plurality of output layers. Eachoutput layer is a coefficient block that corresponds to a fullyconvolutional (FC) down-sample layer.

In some embodiments of the apparatus, the FLI input data is selectedfrom the group including visible FLI microscopy (FLIM) data, nearinfrared (NIR) FLIM data and NIR gated macroscopy FLI (MFLI) data.

In some embodiments, there is provided a method for fluorescencelifetime imaging (FLI). The method includes receiving, by a firstconvolutional layer of a deep neural network (DNN), FLI input data. Themethod further includes receiving, by each intermediate layer of aplurality of intermediate layers, a respective intermediate inputcorresponding to an output of a respective prior layer. The methodfurther includes providing, by each intermediate layer, a respectiveintermediate output related to the received respective intermediateinput; and providing, by an output layer, estimated FLI output datacorresponding to the received FLI input data. The DNN is trained usingsynthetic data.

In some embodiments of the method, the first convolutional layer is athree-dimensional (3D) convolutional layer. The plurality ofintermediate layers includes a 3D residual block, a reshape layer, atwo-dimensional (2D) convolutional layer, and at least one 2D residualblock. The output layer is a fully convolutional (FC) down-sample layer.

In some embodiments of the method, the first convolutional layer is aseparable two-dimensional (2D) convolutional layer. The plurality ofintermediate layers includes a 2D exception block, a 2D convolutionallayer, and a 2D residual block. The output layer is a fullyconvolutional (FC) down-sample layer.

In some embodiments, the method further includes comparing, by adiscriminator network, estimated training output data with trainingsynthetic output data during training. The DNN and the discriminatornetwork correspond to a generative adversarial network (GAN) duringtraining.

In some embodiments, the method further includes receiving, by a secondconvolutional layer, optical property data. The plurality ofintermediate layers includes a concatenate layer. The estimated FLIoutput data is further related to the optical property data.

In some embodiments of the method, the first convolutional layer is athree-dimensional (3D) convolutional layer. The plurality ofintermediate layers includes a second 3D convolutional layer, a reshapelayer, a separable two-dimensional (2D) convolutional layer, and a 2Dexception block. The DNN includes a plurality of output layers. Eachoutput layer is a coefficient block that corresponds to a fullyconvolutional (FC) down-sample layer.

In some embodiments of the method, the FLI input data is selected fromthe group including visible FLI microscopy (FLIM) data, near infrared(NIR) FLIM data and NIR gated macroscopy FLI (MFLI) data

In some embodiments, there is provided a fluorescence lifetime imaging(FLI) deep learning system. The system includes a processor; a memory;input/output circuitry; and a deep neural network (DNN). The DNNincludes a first convolutional layer, a plurality of intermediatelayers, and an output layer. The first convolutional layer is configuredto receive FLI input data. Each intermediate layer is configured toreceive a respective intermediate input corresponding to an output of arespective prior layer. Each intermediate layer is further configured toprovide a respective intermediate output related to the receivedrespective intermediate input. The output layer is configured to provideestimated FLI output data corresponding to the received FLI input data.The DNN is trained using synthetic data.

In some embodiments of the system, the first convolutional layer is athree-dimensional (3D) convolutional layer. The plurality ofintermediate layers includes a 3D residual block, a reshape layer, atwo-dimensional (2D) convolutional layer, and at least one 2D residualblock. The output layer is a fully convolutional (FC) down-sample layer.

In some embodiments of the system, the first convolutional layer is aseparable two-dimensional (2D) convolutional layer. The plurality ofintermediate layers includes a 2D exception block, a 2D convolutionallayer, and a 2D residual block. The output layer is a fullyconvolutional (FC) down-sample layer.

In some embodiments, the system further includes a discriminator networkconfigured to compare estimated training output data with trainingsynthetic output data during training. The DNN and the discriminatornetwork correspond to a generative adversarial network (GAN) duringtraining.

In some embodiments of the system, the DNN further includes a secondconvolutional layer configured to receive optical property data. Theplurality of intermediate layers includes a concatenate layer. Theestimated FLI output data is further related to the optical propertydata.

In some embodiments of the system, the first convolutional layer is athree-dimensional (3D) convolutional layer. The plurality ofintermediate layers includes a second 3D convolutional layer, a reshapelayer, a separable two-dimensional (2D) convolutional layer, and a 2Dexception block. The DNN includes a plurality of output layers. Eachoutput layer is a coefficient block that corresponds to a fullyconvolutional (FC) down-sample layer.

BRIEF DESCRIPTION OF DRAWINGS

The drawings show embodiments of the disclosed subject matter for thepurpose of illustrating features and advantages of the disclosed subjectmatter. However, it should be understood that the present application isnot limited to the precise arrangements and instrumentalities shown inthe drawings, wherein:

FIG. 1 illustrates a functional block diagram of a fluorescence lifetimeimaging (FLI) deep learning system consistent with several embodimentsof the present disclosure;

FIG. 2 illustrates a functional block diagram of one example deep neuralnetwork (DNN) consistent with several embodiments of the presentdisclosure;

FIGS. 3A through 3C illustrate functional block diagrams of anotherexample DNN consistent with several embodiments of the presentdisclosure;

FIG. 4 illustrates a functional block diagram of an examplediscriminator network consistent with several embodiments of the presentdisclosure;

FIG. 5 illustrates a functional block diagram of another example DNNconsistent with several embodiments of the present disclosure;

FIG. 6 illustrates a functional block diagram of another example DNNconsistent with several embodiments of the present disclosure;

FIGS. 7A and 7B illustrate functional block diagrams of another exampleDNN consistent with several embodiments of the present disclosure; and

FIG. 8 a flowchart of FLI deep learning operations according to variousembodiments of the present disclosure.

Although the following Detailed Description will proceed with referencebeing made to illustrative embodiments, many alternatives,modifications, and variations thereof will be apparent to those skilledin the art.

DETAILED DESCRIPTION

Generally, this disclosure relates to fluorescence lifetime imaging(FLI) using deep learning. An FLI deep learning system includes a deepneural network (DNN) configured to receive FLI input data and to provideas output estimated FLI output data. The FLI input data may be capturedusing a time domain technique and/or a frequency domain technique. TheDNN may be trained using synthetic data. In one nonlimiting example, thesynthetic data may be related to image data retrieved from a modifiedNational Institute of Standards and Technology (MNIST) database. Thetraining input data may include a voxel that includes simulationfluorescence decay data, i.e., synthetic data. The voxel includes twospatial dimensions (e.g., pixels) and a time parameter configured tocorrespond to a time point in the decay data. Training the DNN isconfigured to map each spatially located input pixel to training outputdata. Continuing with this example, the training output data may includeone or more corresponding image data sets.

Training the DNN is performed without reliance on user-defined parametersettings and is thus unbiased. An apparatus, method and/or systemconsistent with the present disclosure may be used for analysis ofmicroscopic or macroscopic data and/or wavelengths in a visible or nearinfrared (NIR) range. Utilizing a trained DNN, the FLI analysis may beperformed in real time or close to real time, thus expanding potentialapplications.

Thus, deep learning may utilized in biological applications.

In an embodiment, there is provided an apparatus for fluorescencelifetime imaging (FLI). The apparatus includes a deep neural network(DNN). The DNN includes a first convolutional layer, a plurality ofintermediate layers, and an output layer. The first convolutional layeris configured to receive FLI input data. Each intermediate layer isconfigured to receive a respective intermediate input corresponding toan output of a respective prior layer. Each intermediate layer isfurther configured to provide a respective intermediate output relatedto the received respective intermediate input. The output layer isconfigured to provide estimated FLI output data corresponding to thereceived FLI input data. The DNN is trained using synthetic data.

FIG. 1 illustrates a functional block diagram of a fluorescence lifetimeimaging (FLI) deep learning system 100 consistent with severalembodiments of the present disclosure. FLI deep learning system 100 isconfigured to receive input data 104 and to provide as output estimatedoutput data 106. In some embodiments, input data 104 may include FLIinput data and estimated output data may include estimated FLI outputdata. For example, FLI input data may include, but is not limited to,visible FLI microscopy (FLIM) data, near infrared (NIR) FLIM data andNIR gated macroscopy FLI (MFLI) data. The FLI input data may be capturedusing a time domain technique and/or a frequency domain technique. Inanother example, estimated FLI output data may include, but is notlimited to, short lifetime image data, τ₁, long lifetime image data, τ₂,and fractional amplitude data, A_(R). In some embodiments, input data104 may further include optical properties (OPs) input data (e.g.,absorption and scattering) and estimated output data may include, forexample, depth information, as will be described in more detail below.In some embodiments, the input data 104 may include time-resolvedcompressive sensing data and the estimated output data 106 may includeintensity image data and/or lifetime image data.

FLI deep learning system 100 includes a deep neural network (DNN) 102.In some embodiments, DNN 102 corresponds to a convolutional neuralnetwork (CNN). FLI deep learning system 100 may further include trainingcircuitry 120 and a training data store 122. For example, trainingcircuitry 120 and the training data store 122 may be included duringtraining DNN 102. Training data store 122 may be configured to store oneor more training data sets 131-1, . . . , 131-m. Each training data set,e.g., training data set 131-1, may include training input data 130 andcorresponding training output data 132. In an embodiment, the trainingdata sets correspond to synthetic data. In other words, in thisembodiment, the training data sets may not include actual FLI input dataand corresponding FLI output data. It may be appreciated that generatingor acquiring synthetic data for training may be relatively easier thanacquiring or generating actual FLI data sets. In one nonlimitingexample, the synthetic data may be related to image data retrieved froma modified National Institute of Standards and Technology (MNIST)database. Each voxel may be simulated using the MNIST database to obtainspatial maps of pixels (e.g., 28×28) and to subsequently generatefluorescence decays (Γ(t)) at each nonzero pixel location using abiexponential model convolved with an experimental instrument responsefunction (IRF):Γ(t)=IRF(t)*[A _(R) e ^(−t/τ) ¹ +(1−A _(R))e ^(−t/τ) ² ]Thus, in some embodiments, data inputs have dimension 28×28×t.

During training, training circuitry 120 may be configured to providetraining input data 130 to DNN 102 and to receive estimated trainingoutput data 133 from DNN 102. Training circuitry 120 may be furtherconfigured to compare the estimated training output data 133 to thetraining output data 132 that corresponds to the training input data130. Training circuitry 120 may be further configured to adjust one ormore DNN parameters 134 of DNN 102 based, at least in part, on thecomparison. The DNN parameters 134 may include, for example, filtervalues, weights, etc. For example, training circuitry 120 may beconfigured to minimize a loss function based, at least in part, on thecomparison of the estimated training output data 133 and correspondingtraining output data 132.

In some embodiments, training circuitry 120 may include a discriminatornetwork 124. In these embodiments, DNN 102 and discriminator network 124may correspond to a generative adversarial network (GAN). During GANtraining, discriminator network 124 is configured to receive estimatedtraining output data 133 and to determine whether the received estimatedtraining output data is “real” or “fake”. Discriminator network 124 maybe utilized during training of a generative network, e.g., DNN 102, in agenerative adversarial network (GAN). In other words, the output of thediscriminator network 124 is utilized by training circuitry 120 toadjust DNN parameters 134 such that the estimated training output data133 from the DNN 102 cannot be differentiated from ground truth trainingoutput data 132.

FLI deep learning system 100 includes a processor 112, memory 114, andinput/output (I/O) circuitry 116. Processor 112 may include, but is notlimited to, a single core processing unit, a multicore processor, agraphics processing unit, etc. Memory 114 may be configured to store oneor more of DNN 102, training circuitry 120, training data store 122and/or discriminator 124. I/O circuitry 116 may be configured to receiveinput data 104 and/or to provide estimated output data 106.

Thus, FLI deep learning system 100 includes DNN 102 configured toreceive input data 103 and to provide estimate output data 106. DNN 102may be trained using training data sets 131-1, . . . , 131-m thatinclude synthetic training data. In some embodiments, training maycorrespond to a GAN technique and, in these embodiments, trainingcircuitry 120 may include discriminator 124. After training, DNN 102 maybe configured to receive FLI input data and to provide correspondingestimated FLI output data, as described herein.

In some embodiments, a selected architecture of DNN 102 may be relatedto a particular application of the DNN 102. For example, thearchitecture of DNN 102 may be selected based, at least in part, on oneor more of characteristics of the input data, characteristics of theestimated output data, a target processing time, and/or capacity ofprocessor 112. As used herein, DNN architecture corresponds to arespective type of each functional block and configuration of thefunctional blocks (i.e., layers) of the corresponding DNN. As usedherein, “block”, “functional block” and “layer” are usedinterchangeably.

FIG. 2 illustrates a functional block diagram of one example DNN 202consistent with several embodiments of the present disclosure. ExampleDNN 202 is configured to receive FLI input data 204 and to provide FLIoutput data as output. Example DNN 202 includes a three-dimensional (3D)convolution block (Conv3D) 210, a 3D residual block (ResBlock3D) 212, areshaping block (Reshape) 214, a two dimensional (2D) convolution block(Conv2D) 216, and one or two 2D residual blocks (ResBlock2D) 218-1,218-2. Conv3D 210, ResBlock3D 212, Reshape 214, Conv2D 216, andResBlock2D 218-1, 218-2 correspond to a shared branch 203.

Example DNN 202 further includes a plurality of FC (fully convolutional)down-sample blocks 220-1, 220-2, 220-3. Each FC down-sample block isconfigured to receive an output from ResBlock2D 218-2 and to provide asoutput a respective estimated FLI output data. In this example, a firstFC down-sample block 220-1 is configured to provide as output 206-1short lifetime image data, τ₁, a second FC down-sample block 220-2 isconfigured to provide as output 206-2 long lifetime image data, τ₂, anda third FC down-sample block 220-3 is configured to provide as output206-3 fractional amplitude data, A_(R). Each FC down-sample block is220-1, 220-2, 220-3 is included in a respective separate branch 205-1,205-2, 205-3, coupled to an output of the shared branch 203.

Example DNN 202 architecture corresponds to a 3D convolutional neuralnetwork (CNN) architecture. Example DNN 202 may be termed fluorescencelifetime imaging network (FLI-Net) and is configured to process datasetsthat may be acquired by fluorescence imaging systems. DNN 202 may beconfigured to provide, for example, one or more lifetime map(s),associated quantities (i.e., lifetime species, E %, or fractionalamplitude of lifetime components, such as FRET donor fraction [FD %]).In one nonlimiting example, DNN 202 may be trained to reconstruct mono-and biexponential FLI for one or more classes of experiments that may beencountered in the field. Such experiments may include, but are notlimited to, in vitro FLI microscopy (FLIM) in cultured cells in thenear-infrared (NIR) and visible range, and/or in vivo NIR macroscopicFLI (MFLI) in live intact mice. For each of these cases, the data usedin the training datasets may be simulated over relatively wide lifetimebounds configured to encompass those present in the application ofinterest. DNN 202 may be trained relatively efficiently using asynthetic data generator configured to support FLI reconstruction withexperimental datasets not used during training. DNN 202 is configured tobe relatively accurate over a relatively large range of lifetimes(including those close to the instrument response) and may providesuperior performances in photon-starved conditions. In some embodiments,DNN 202 may be configured to process experimental fluorescent decaysacquired by either time-correlated single-photon counting (TCSPC)-based(FLIM datasets) or gated intensity charged-coupled device (ICCD)-based(MFLI datasets) instruments. In one nonlimiting example, DNN 202 may beconfigured to quantify whole-body dynamic lifetime-based FRET occurrencein a live intact animal at a frame rate of approximately 80 ms(milliseconds) per full whole-body image. However, this disclosure isnot limited in this regard.

DNN 202 may be configured to receive time-resolved and spatiallyresolved fluorescence decay inputs as 3D data cube (x, y, t).Biexponential parameters (e.g., lifetimes, τ₁ and τ₂, and fractionalamplitude, A_(R)) may then be independently estimated at each pixel andconfigured to be provided in output images of the same dimension as theinput (x, y). The DNN 202 architecture generally includes two portions:the shared branch 203 and separate branches 205-1, 205-2, 205-3. Theshared branch 203 is configured for temporal feature extraction. Thesubsequent separate branches are configured for simultaneousreconstruction of short lifetime (τ₁), long lifetime (τ₂), andfractional amplitude of the short lifetime (A_(R)), respectively. Afirst convolutional layer, i.e., Conv3D 210, is configured to maximizespatially independent feature extraction along each temporal pointspread function (TPSF). In one nonlimiting example, Conv3D layer 210 maybe implemented with kernel size of (1×1×10) configured to mitigate apotential introduction of unwanted artifacts dependent on neighboringpixel information in the spatial dimensions (x and y) during trainingand inference. ResBlock3D 212 may be configured with a reduced kernellength. ResBlock3D 212 is configured to further extract temporalinformation while reaping the benefits obtained through residuallearning (e.g., elimination of vanishing gradients, no overall increasein computational complexity or parameter count, etc.). After performingthe common features of the whole input in the shared branch 203, DNN 202splits into the 3 dedicated fully convolutional branches 205-1, 205-2,205-3 configured to estimate the individual lifetime-based parameters ofinterest, i.e., short lifetime (τ₁), long lifetime (τ₂), and fractionalamplitude of the short lifetime (A_(R)). In each of these branches205-1, 205-2, 205-3, a sequence of convolution operations may beemployed for down-sampling to the intended 2D image.

Thus, DNN 202 architecture may include the shared branch 203 focused onspatially independent temporal feature extraction and a subsequent3-junction split for independent reconstruction of τ₁, τ₂, and A_(R)images simultaneously. In one nonlimiting example, within the sharedbranch 203, spatially independent convolutions along time with kernelsize of (1×1×10) may correspond to a first layer (i.e., Conv3D 210) tomaximize TPSF feature extraction. In another nonlimiting example, acorresponding stride of k=(1,1,5), may be implemented, configured toreduce parameter count and increase computational speed, while resultingin no observable decrease in performance. ResBlock3D 212, a residualblock, may have a kernel size of (1×1×5), and is configured to furtherextract time-domain information. To obtain image reconstruction of size(x×y) via a sequence of down-sampling, a transformation from 4D to 3Dwas implemented. Thus, after ResBlock3D 212 (output of x×y×n×50), atensor may be reshaped to dimension (x×y×(n×50)) by Reshape block 214,where n corresponds to a scalar value dependent on the number of TPSFtime points and the network hyperparameters.

The value of n may be determined as:

$P = {\frac{F_{L0}}{2}\left( {n_{TP}\% S} \right)}$ and$n = \left( {\frac{\left( {n_{TP} - F_{L0} + P} \right)}{S} + 1} \right)$where n_(TP), P, F_(L0) and S denote the number of time points, padding,filter length along the temporal direction of Conv3D 210 layer (e.g.,length of 10), and the corresponding stride value used in the firstconvolutional layer (e.g., value of 5), respectively. Conv2D 216 layer,a convolutional layer having size (1×1) possessing 256 filters followedby ResBlock2D 218-1, 218-2, a subsequent residual block coupletpossessing size (1×1) were implemented before the tri-reconstructionjunction. The (1×1) size of these 2D convolutional filters areconfigured to maintain spatially independent feature extraction.

In one nonlimiting example, a total of 10,000 TPSFS voxels were usedduring training (8,000) and validation (2,000), along with a batch sizedependent on the target input length along time (32 for NIR, 20 forvisible). Continuing with this example, MSE may be set as the lossfunction for each branch. The DNN 202 may be trained, for example, at250 epochs using a NVIDIA TITAN Xp GPU. This training time may vary,based, at least in part, on TPSF length, e.g., may range between 50 sand 80 s per epoch (for voxels possessing 160 and 256 time points,respectively).

Thus, DNN 202 may be configured to provide FLI image information, basedon deep learning (DL) and configured to quantify fluorescence decayssimultaneously over a whole image and at relatively fast speeds, i.e.,at or approaching real time. The deep neural network architecture may bedesigned and trained for different classes of experiments, includingvisible FLI and near-infrared (NIR) FLI microscopy (FLIM) and MR gatedmacroscopy FLI (MFLI). DNN 202 may be configured to output,quantitatively, the spatially resolved lifetime-based parameters thatmay be typically employed in the field. Utility of the FLI-Net frameworkmay be validated, for example, by performing quantitative microscopicand preclinical lifetime-based studies across the visible and NIRspectra, as well as across the 2 main data acquisition technologies. DNN202 may thus be configured to relatively accurately quantify complexfluorescence lifetimes in cells and, in real time, in intact animalswithout user-defined parameter settings

FIGS. 3A through 3C illustrate functional block diagrams of anotherexample DNN. Turning first to FIG. 3A, FIG. 3A illustrates a functionalblock diagram of another example DNN 302 consistent with severalembodiments of the present disclosure. Example DNN 302 is configured toreceive FLI input data 304 and to provide FLI output data as output.Similar to DNN 202 of FIG. 2 , DNN 302 includes a shared branch 303 anda plurality of separate branches 305-1, 305-2, 305-3, coupled to anoutput of the shared branch 303. Example DNN 302 and shared branch 303includes a separable two-dimensional (2D) convolution block (SeparableConv2D) 310, a 2D exception block (XceptionBlock2D) 312, a twodimensional (2D) convolution block (Conv2D) 316, and one or two 2Dresidual blocks (ResBlock2D) 318-1, 318-2.

Example DNN 302 further includes a plurality of FC (fully convolutional)down-sample blocks 320-1, 320-2, 320-3. Each FC down-sample block is320-1, 320-2, 320-3 is included in a respective separate branch 305-1,305-2, 305-3. Each FC down-sample block is configured to receive anoutput from ResBlock2D 318-2 and to provide as output a respective FLIoutput data. In this example, a first FC down-sample block 320-1 isconfigured to provide as output 306-1 short lifetime image data τ₁, asecond FC down-sample block 320-2 is configured to provide as output306-2 long lifetime image data τ₂, and a third FC down-sample block320-3 is configured to provide as output 306-3 fractional amplitudeA_(R).

Example DNN 302 architecture corresponds to a 2D convolutional neuralnetwork (CNN) architecture. Example DNN 302 may be termed fluorescencelifetime imaging (microscopic) network (FLIM-Net) and is configured toprocess datasets that may be acquired by fluorescence imaging systems.

FIG. 3B illustrates a functional block diagram 330 of a two-dimensional(2D) exception block, e.g., XceptionBlock2D 312 of FIG. 3A. 2D exceptionblock 330 is configured to receive an output from the separable 2Dconvolution block Separable Conv2D 310 and to provide an input to the 2Dconvolution block (Conv2D) 316 of FIG. 3A.

2D exception block 330 includes a first 2D separable convolution block(Separable Conv2D) 332-1, a block 334 that includes batch normalization(BN) and a rectified linear unit (ReLU), a second 2-D separableconvolution block 332-2, a batch normalization (BN) block 336, an adder338, and an rectified linear unit (ReLU) 340. The adder 338 isconfigured to add an output from BN block 336 and an input correspondingto the output from Separable Conv2D block 310. An output from BN block336 is then provided to ReLU block 340 and an output from ReLU block 340corresponds to the output of 2D exception block 330 that is provided toConv2D block 316 of FIG. 3A.

FIG. 3C illustrates a functional block diagram 350 of a fullyconvolutional (FC) down-sample block, e.g., FC down-sample blocks 320-1,320-2, 320-3 of FIG. 3A. FC down-sample block 350 is configured toreceive an output from the 2D residual block ResBlock2D 318-2 and toprovide lifetime imaging output data as output 306-x. FC down-sampleblock 350 includes three two-dimensional convolutional blocks (Conv2D)352-1, 352-2, 352-3, separated by intermediate blocks 354-1, 354-2. Eachintermediate block 354-1, 354-2 includes a batch normalization (BN)function and a rectified linear unit (ReLU). FC down-sample block 350further includes an ReLU block 356 coupled to an output of Conv2D block352-3 and configured to provide FLI output data. FLI output dataincludes, in this example, short lifetime image data τ₁, long lifetimeimage data τ₂, and fractional amplitude A_(R).

DNN 302 may be configured to receive time-resolved and spatiallyresolved fluorescence decay inputs as 3D data cube (x, y, t).Biexponential parameters (e.g., lifetimes, τ₁ and τ₂, and fractionalamplitude, A_(R)) may then be independently estimated at each pixel andconfigured to be provided in output images of the same dimension as theinput (x, y). The DNN 302 architecture generally includes two portions:the shared branch 303 and separate branches 305-1, 305-2, 305-3. Theshared branch 303 is configured for temporal feature extraction. Thesubsequent separate branches are configured for simultaneousreconstruction of short lifetime (τ₁), long lifetime (τ₂), andfractional amplitude of the short lifetime (A_(R)), respectively.

A first convolutional layer, i.e., SeparableConv2D 310, is configured tomaximize spatially independent feature extraction along each temporalpoint spread function (TPSF). In one nonlimiting example,SeparableConv2D 310 may be implemented with kernel size of (1×1)configured to mitigate a potential introduction of unwanted artifactsdependent on neighboring pixel information in the spatial dimensions (xand y) during training and inference. SeparableConv2D 310 may be furtherimplemented with 512 filters. The first convolutional layer may befollowed by an exception block, e.g., XceptionBlock2D, implemented withkernel size of (1×1) with 512 filters. The exception block may include asequence of separable 2D convolution, batch normalization and ReLUoperations. In one nonlimiting example, the sequence of operations maybe configured as example XceptionBlock2D block 330 of FIG. 3B. TheXceptionBlock2D 312 may be followed by a 2D convolutional block withkernel size of (1×1) with 256 filters. ResBlock2D 318-1, 318-2, residualblocks, may have a kernel size of (1×1) with 256 filters, and areconfigured to further extract time-domain information.

After performing the common features of the whole input in the sharedbranch 303, DNN 302 splits into the 3 dedicated fully convolutionalbranches 305-1, 305-2, 305-3 configured to estimate the individuallifetime-based parameters of interest, i.e., short lifetime (τ₁), longlifetime (τ₂), and fractional amplitude of the short lifetime (A_(R)).In each of these branches 305-1, 305-2, 305-3, a sequence ofconvolution, batch normalization and ReLU operations may be employed fordown-sampling to the intended 2D image. In one nonlimiting example, thesequence of operations may be configured as example FC down-sample block350 of FIG. 3C.

Thus, DNN 302 architecture may include the shared branch 303 focused onspatially independent temporal feature extraction and a subsequent3-junction split for independent reconstruction of τ₁, τ₂, and A_(R)images simultaneously. DNN 302 architecture may thus correspond to a 2Darchitecture optimized for FLIM. The 2D convolutional architecture isconfigured to be relatively more computationally efficient compared tothe 3D convolutional architecture of, for example, FIG. 2 . DNN 302 maybe an order of magnitude faster (both training and use) compared to DNN202. DNN 302 may be implemented on a general purpose processor ratherthan a graphics processing unit for processing conventional FLIM data.

Thus, DNNs including 2D and 3D convolutional layers may be implementedin FLI deep learning system 100.

FIG. 4 illustrates a functional block diagram of an examplediscriminator network 400 consistent with several embodiments of thepresent disclosure. Discriminator network 400 is one example ofdiscriminator 124 of FIG. 1 . Discriminator network 400 is configured toreceive training output data 132 as FLI output data and to generate abinary output 414 that corresponds to “real” or “fake”. Discriminatornetwork 400 includes a first Conv2D block 402-1, a first intermediateblock 404-1 that includes a BN function and a leaky ReLU, a secondConv2D block 402-2, and a second intermediate block 404-2. Discriminator400 further includes a flatten block 406, a first dense block 408-1, athird intermediate block 404-3, a second dense block 408-2, a dropoutblock 410, and a sigmoid block 412. Input to the discriminator network400 is received by the first Conv2D block 402-1 and the binary output414 from the discriminator network 400 is provided by the sigmoid block412.

Thus, discriminator network 400 may be utilized during training of DNN102 of FIG. 1 when training corresponds to a GAN technique, as describedherein. In one nonlimiting example, discriminator network 400 may beused with example DNN 302. However, this disclosure is not limited inthis regard.

FIG. 5 illustrates a functional block diagram of another example DNN 502consistent with several embodiments of the present disclosure. ExampleDNN 502 is configured to receive, as input, FLI input data 504-1 andoptical properties (OPs) data 504-2, and to provide, as output,spatially resolved lifetime (τ) data 506-1 and depth (Z) data 506-2.Example DNN 502 includes two input branches 505-1, 505-2. A first inputbranch 505-1 is configured to receive FLI input data 504-1 and a secondinput branch 505-2 is configured to receive OPs data 504-2. The firstinput branch 505-1 generally corresponds to blocks 210 through 216 ofFIG. 2 . Thus, the first input branch 505-1 includes a Conv3D block 510,a ResBlock3D block 512, a Reshape block 514, a first Conv2D block 516-1.The first input branch 505-1 further includes a second Conv2D block516-2. The second input branch 505-2 includes a third Conv2D block516-3, a fourth Conv2D block 517-1, and a fifth Conv2D block 517-2.

Example DNN 502 further includes a concatenate block 523 configured toreceive respective outputs from the first and second input branches505-1, 505-2. Example DNN 502 further includes a separable Conv2D block525, a first ResBlock2D block 518-1, and a second ResBlock2D block518-2.

Example DNN 502 further includes a plurality of FC (fully convolutional)down-sample blocks 520-1, 520-24. Each FC down-sample block isconfigured to receive an output from ResBlock2D 518-2 and to provide asoutput a respective output data. In this example, a first FC down-sampleblock 220-1 is configured to provide as output 506-1 spatially resolvedlifetime (t) data, and a second FC down-sample block 520-2 is configuredto provide as output 506-2, depth (Z) data.

DNN 502 may correspond to a macroscopic fluorescence lifetime imaging(MFLI) topography computational framework. In one nonlimiting example,DNN 502 may be configured to retrieve the depth of fluorescentinclusions deeply seated in bio-tissues. DNN 502 is configured toleverage depth-resolved information included in time-resolvedfluorescence data sets coupled with retrieval of in situ opticalproperties as obtained via spatial frequency domain imaging (SFDI). DNN502 architecture is configured to receive optical properties (OPs) andtime-resolved fluorescence decays as input and to provide simultaneousretrieval of lifetime maps and depth profiles. Experimental resultsdemonstrate that DNN 502 can retrieve the depth of fluorescenceinclusions, when coupled with optical properties estimation, withrelatively high accuracy. It is contemplated that a DNN architecturecorresponding to DNN 502 may find utility in applications such asoptical-guided surgery.

Thus, DNN 502 may be configured to provide optical properties-correctedwide-field macroscopic fluorescence topography. DNN 502 may beconfigured to support end-to-end DL workflow inputs wide-field OPs maps(absorption (μ_(a)) and reduced scattering (μ_(s)′) as obtained viaSFDI) and wide-field time-resolved fluorescence data; and outputs bothlifetime (τ) and 2D depth profiles (z) maps at the same resolution asthe inputs.

FIG. 6 illustrates a functional block diagram of another example DNN 602consistent with several embodiments of the present disclosure. ExampleDNN 602 is configured to perform spectral and lifetime fluorescenceunmixing, as will be described in more detail below. Example DNN 602 isconfigured to receive FLI input data 604 and to provide a plurality ofcoefficients 606-1, 606-2, . . . , 606-N as output. Each coefficientcorresponds to a respective fluorescence concentration.

Example DNN 602 includes a first three-dimensional (3D) convolutionblock (Conv3D) 610-1, a second Conv3D block 610-2, a reshaping block(Reshape) 614, a two dimensional (2D) separable convolution block(Separable Conv2D) 611, and a 2D exception block (XceptionBlock2D) 613.In some embodiments, XceptionBlock2D 613 corresponds to 2D exceptionblock 330 of FIG. 3B.

Example DNN 602 further includes a plurality of coefficient blocks620-1, 620-2, . . . , 620-N, that may each correspond to an FCdown-sample block. Each coefficient block is configured to receive anoutput from XceptionBlock2D 613 and to provide as output respectivecoefficient data, c₁, c₂, . . . , c_(N). In some embodiments, eachcoefficient block 620-1, 620-2, . . . , 620-N corresponds to FCdown-sample block 350 of FIG. 3C.

Hyperspectral fluorescence lifetime imaging allows for the simultaneousacquisition of spectrally resolved temporal fluorescence emissiondecays. The acquired multidimensional data set may enable simultaneousimaging of a plurality of fluorescent species for a comprehensivemolecular assessment of biotissues. Spectral overlap between theconsidered fluorescent probes and potential bleed-through may be presentand should be considered to enable quantitative imaging. Example DNN 602may be configured to implement a deep learning-based fluorescenceunmixing routine, capable of quantitative fluorophore unmixing bysimultaneously using both spectral and temporal signatures. DNN 602 maybe configured to perform fluorophore unmixing by leveraging bothspectral and lifetime contrast concomitantly.

DNN 602 may be configured to retrieve the spatially resolved abundancecoefficients associated with a sample's fluorophores from the spectrallyresolved fluorescence decay measurements within the context ofHyperspectral Macroscopic Fluorescence Lifetime Imaging (HMFLI). DNN 602may be configured to leverage a single-pixel strategy to concurrentlyacquire 16 spectrally resolved FLI channels over relatively large fieldof views (FOV). Through the use of Deep Learning (DL), HMFLI may becapable of probing nanoscale biomolecular interactions across largefields of view (FOV) at resolutions as high as 128×128 within minutes.DL may reduce the processing time for its inverse solving procedure,yielding intensity and lifetime reconstructions in a single framework,through usage of simulated training data mimicking the single-pixel datageneration.

DNN 602 may be trained generally as described herein using syntheticdata, e.g., a binary handwritten number dataset EMINST for assignment ofspatially-independent random variables of TPSF (Γ(t)) as:Γ^(λ)(t)=I×IRF ^(λ)(t)*[a ₁ ^(λ) e ^(−t/τ) ¹ +a ₂ ^(λ) e ^(−t/τ) ² + . .. +a _(n) ^(λ) e ^(−t/τ) ^(n) ]where IRF^(λ)(t) corresponds to the instrument response function, a_(n)^(λ) the relative spectral brightness of the n^(th) fluorophore at thewavelength λ, τ_(r), to the lifetime value of the n^(th) fluorescentspecies with associated relative abundance coefficients (c^(n)), and Iis an intensity scalar corresponding to the overall photon counts to bedetected. The output data may then correspond to abundance coefficients,c₁, c₂, . . . , c_(n). In one nonlimiting example, variables used duringspatially-independent generation of TPSFs may be assigned random values,with the values included in a range of typical values for the particularimaging technology, e.g., NIR fluorescence imaging. Thus, duringtraining, input to the DNN 602 may correspond to a four-dimensional (4D)voxel (16×16×wavelength channel×time) that includes simulated (i.e.,synthetic) hyperspectral fluorescence decay data (t corresponding totime points). The DNN 602 may then be trained to map each spatiallocation's spectral and lifetime data to N 16×16 images, with each imagecontaining a respective fluorescence concentration of a particularfluorophore. After training, the DNN 602 may be used to analyze data ofany spatial dimensions.

DNN 602 architecture is configured to prioritize extraction of temporalinformation while mitigating a computational burden associated withprocessing simultaneously 16 spectral channels-worth of TPSF data. Inone nonlimiting example, the first 3D convolutional layer Conv3D 610-1may have kernel size of (1×1×16), a corresponding stride of k=(1×1×8)and 64 filters. In another nonlimiting example, the second 3Dconvolutional layer Conv3D 610-2 may have kernel size of (1×1×8), acorresponding stride of k=(1×1×4) and 64 filters. The stride values areconfigured to support a reduction in parametric size within the earlylayers. The output from the Conv3D layers may be transformed (i.e.,reshaped) into 2D (x×y, CH_(#)×64) by Reshape 614. The 2D separableconvolution layer may have a kernel size of 1×1 that may correspond to amore computationally friendly alternative for spatially-independenttemporal and spectral feature extraction. XceptionBlock operations(i.e., residual blocks with 1×1 separable convolution operations) may beconfigured to exploit residual learning while supportingspatially-independent temporal and spectral feature extraction.“CoefficientBlocks” correspond to individual branches that include a setof 2D convolution operations intercepted by batch normalization andactivation layers, with each branch meant to focus on features relevantfor fluorophore-specific abundance coefficient retrieval. These blocksare configured to facilitate seamless architecture modification forretrieval of N number of targeted fluorophores.

Thus, DNN 602 may be configured to implement quantitative fluorophoreunmixing by simultaneously using both spectral and temporal signatures.

FIGS. 7A and 7B illustrate functional block diagrams of another exampleDNN consistent with several embodiments of the present disclosure.Turning first to FIG. 7A, FIG. 7A illustrates a functional block diagramof another example DNN 702 consistent with several embodiments of thepresent disclosure. Example DNN 702 is configured to receive timeresolved compressive sensing (CS) input data 704 and to provide lifetimeimage output data 706-2 and intensity image output data 706-1 as output.

Example DNN 702 includes a first combination block 710-1 that includes aone dimensional (1D) convolution function (Conv1D), a BN function (BN),and a ReLU. Example DNN 702 further includes a transpose block 712 and afirst branch 713-1 and a second branch 713-2 output from the transposeblock 712. The first branch 713-1 includes a second combination block710-2 that includes Conv1D, BN and ReLU. The first branch 713-1 furtherincludes a reshape block (Reshape) 714-2, an intermediate block 720 thatincludes a separable Conv2D and a ReLU, a residual block (ResBlock)716-2, and two reconstruction blocks (ReconBlock) 718-2, 718-3. Theoutput 706-1 of the first branch 713-1 corresponds to lifetime imagedata

The second branch 713-2 includes a reshape block (Reshape) 714-1, aresidual block (ResBlock) 716-1, and a first reconstruction block(ReconBlock) 718-1. The first reconstruction block ReconBlock 718-1includes three intermediate blocks 719-1, 719-2, 719-3. Eachintermediate block 719-1, 719-2, 719-3 includes a two-dimensionalconvolution function (Conv2D) and a ReLU. The output 706-2 of the secondbranch 713-2 corresponds to intensity image data.

FIG. 7B illustrates a functional block diagram 730 of a residual block,e.g., ResBlock 716-1, 716-2 of FIG. 7A. Residual block 730 is configuredto receive an input 732 from a prior block and to provide an output 738to a subsequent block of FIG. 7A. Residual block 730 includes a first 2Dconvolutional block (Conv2D) 734-1, a first rectified linear unit (ReLU)736-1, a second Conv2D block 734-2, an adder 736, and a second ReLU736-2. The first Conv2D block 734-1 and the adder 736 are configured toreceive the input 732 to the residual block 730. The adder is furtherconfigured to receive an output of the second Conv2D block 734-2. Thesecond ReLU 736-2 is configured to provide the output 738.

Single pixel imaging frameworks may facilitate the acquisition ofrelatively high-dimensional optical data in biological applications withphoton starved conditions. Single pixel imaging frameworks may haverelatively slow acquisition times and low pixel resolution. DNN 702 maybe configured to implement compressed sensing at relatively highcompression (NetFLICS-CR). DNN 702 may thus enable in vivo applicationsat enhanced resolution, acquisition and processing speeds. DNN 702 maybe configured to produce intensity and lifetime reconstructions at128×128 pixel resolution over 16 spectral channels while using at most1% of the measurements. Acquisition times may thus be on the order of ˜3minutes at 99% compression. For example, DNN 702 may be configured forin vivo monitoring of lifetime properties and drug uptake.

DNN 702 may be configured with a relatively fast processing speed forsimultaneous intensity and lifetime retrieval, reconstructing 128×128pixels while using compression ratios (CRs) of up to 99% to reduceacquisition time. DNN 702 has a three branched structure, with a commonsegment (blocks 710-1 and 712) that derives into separate intensity713-2 and lifetime 713-1 branches. To increase the training robustnesson the lifetime branch 713-1, 2D Separable Convolutional blocks areconfigured to better extract the lifetime features along the timedimension of the data. The intensity branch 713-2 includes a singleResBlock 716-1 and a ReconBlock 718-1 that yields the reconstructed128×128 intensity image 706-2. The lifetime branch 713-1 includesResBlock 716-2, a double ReconBlock structure 718-2, 718-3 and a 1Dconvolutional layer 710-1 continued by a 2D separable convolutionalblock 720. The 1D convolutional layers are followed by batchnormalization and ReLU activations. For training, the synthetic datasetmay be used as the spatial model to generate 128×128 pixels images.Then, at each pixel, fluorescence time-domain data may be simulatedusing an exponential fluorescent decay model convolved with an HMFLIinstrumental response function (IRF).

Thus, DNN 702 may be configured to reduce the acquisition time ofhigh-dimensional SP-MFLI optical molecular data, while simultaneouslyproducing 128×128 hyperspectral intensity and lifetime maps.

Thus, an FLI deep learning system includes a deep neural network (DNN)configured to receive FLI input data and to provide as output estimatedFLI output data. The DNN may be trained using synthetic data. Trainingthe DNN is configured to map each spatially located input pixel totraining output data. The training output data may include one or morecorresponding image data sets.

Training the DNN is performed without reliance on user-defined parametersettings and is thus unbiased. An apparatus, method and/or systemconsistent with the present disclosure may be used for analysis ofmicroscopic or macroscopic data and/or wavelengths in a visible or nearinfrared (NIR) range. Utilizing a trained DNN, the FLI analysis may beperformed in real time or close to real time, thus expanding potentialapplications. Thus, deep learning may utilized in biologicalapplications.

FIG. 8 a flowchart 800 of FLI deep learning operations according tovarious embodiments of the present disclosure. In particular, theflowchart 800 illustrates fluorescence lifetime imaging (FLI) operationsconfigured to provide estimated output data based, at least in part, onFLI input data. The operations may be performed, for example, by FLIdeep learning system (e.g., DNN 102 and/or training circuitry 120) ofFIG. 1 .

Operations of this embodiment may begin with receiving, by a firstconvolutional layer of a deep neural network (DNN), FLI input data atoperation 802. Operation 804 includes receiving, by each intermediatelayer of a plurality of intermediate layers, a respective intermediateinput corresponding to an output of a respective prior layer. Operation806 includes providing, by each intermediate layer, a respectiveintermediate output related to the received respective intermediateinput. Operation 808 includes providing, by an output layer, estimatedFLI output data corresponding to the received FLI input data. The DNN istrained using synthetic data.

In some embodiments, FLI deep learning operations may further includecomparing, by a discriminator network, estimated training output datawith training synthetic output data during training at operation 810.The DNN and the discriminator network correspond to a generativeadversarial network (GAN) during training. In some embodiments, FLI deeplearning operations may further include receiving, by a secondconvolutional layer, optical property data at operation 812. Theplurality of intermediate layers includes a concatenate layer. Theestimated FLI output data is further related to the optical propertydata.

Thus, estimated output data may be provided based, at least in part, onFLI input data.

As used in any embodiment herein, the term “logic” may refer to an app,software, firmware and/or circuitry configured to perform any of theaforementioned operations. Software may be embodied as a softwarepackage, code, instructions, instruction sets and/or data recorded onnon-transitory computer readable storage medium. Firmware may beembodied as code, instructions or instruction sets and/or data that arehard-coded (e.g., nonvolatile) in memory devices.

“Circuitry”, as used in any embodiment herein, may comprise, forexample, singly or in any combination, hardwired circuitry, programmablecircuitry such as computer processors comprising one or more individualinstruction processing cores, state machine circuitry, and/or firmwarethat stores instructions executed by programmable circuitry. The logicmay, collectively or individually, be embodied as circuitry that formspart of a larger system, for example, an integrated circuit (IC), anapplication-specific integrated circuit (ASIC), a system on-chip (SoC),desktop computers, laptop computers, tablet computers, servers, smartphones, etc.

Memory 112 may include one or more of the following types of memory:semiconductor firmware memory, programmable memory, non-volatile memory,read only memory, electrically programmable memory, random accessmemory, flash memory, magnetic disk memory, and/or optical disk memory.Either additionally or alternatively system memory may include otherand/or later-developed types of computer-readable memory.

Embodiments of the operations described herein may be implemented in acomputer-readable storage device having stored thereon instructions thatwhen executed by one or more processors perform the methods. Theprocessor may include, for example, a processing unit and/orprogrammable circuitry. The storage device may include a machinereadable storage device including any type of tangible, non-transitorystorage device, for example, any type of disk including floppy disks,optical disks, compact disk read-only memories (CD-ROMs), compact diskrewritables (CD-RWs), and magneto-optical disks, semiconductor devicessuch as read-only memories (ROMs), random access memories (RAMs) such asdynamic and static RAMs, erasable programmable read-only memories(EPROMs), electrically erasable programmable read-only memories(EEPROMs), flash memories, magnetic or optical cards, or any type ofstorage devices suitable for storing electronic instructions.

The terms and expressions which have been employed herein are used asterms of description and not of limitation, and there is no intention,in the use of such terms and expressions, of excluding any equivalentsof the features shown and described (or portions thereof), and it isrecognized that various modifications are possible within the scope ofthe claims. Accordingly, the claims are intended to cover all suchequivalents.

Various features, aspects, and embodiments have been described herein.The features, aspects, and embodiments are susceptible to combinationwith one another as well as to variation and modification, as will beunderstood by those having skill in the art. The present disclosureshould, therefore, be considered to encompass such combinations,variations, and modifications.

What is claimed is:
 1. An apparatus for fluorescence lifetime imaging(FLI), the apparatus comprising: a deep neural network (DNN) comprising:a first convolutional layer configured to receive FLI input data; aplurality of intermediate layers, each intermediate layer configured toreceive a respective intermediate input corresponding to an output of arespective prior layer, each intermediate layer further configured toprovide a respective intermediate output related to the receivedrespective intermediate input; and an output layer configured to provideestimated FLI output data corresponding to the received FLI input data,the output layer is a fully convolutional (FC) down-sample layercomprising: a first two-dimensional (2D) convolutional block; a firstintermediate block coupled to an output of the first 2D convolutionalblock; a second 2D convolutional block coupled to an output of the firstintermediate block; a second intermediate block coupled to an output ofthe second 2D convolutional block; a third 2D convolutional blockcoupled to an output of the second intermediate block; and a rectifiedlinear unit block coupled to an output of the third 2D convolutionalblock; wherein the first intermediate block and the second intermediateblock each comprise a batch normalization function and a rectifiedlinear unit; wherein the DNN is trained using synthetic data.
 2. Theapparatus of claim 1, wherein the first convolutional layer is athree-dimensional (3D) convolutional layer, the plurality ofintermediate layers comprises a 3D residual block, a reshape layer, atwo-dimensional (2D) convolutional layer, and at least one 2D residualblock.
 3. The apparatus of claim 1, wherein the first convolutionallayer is a separable two-dimensional (2D) convolutional layer; theplurality of intermediate layers comprises a 2D exception block, a 2Dconvolutional layer, and a 2D residual block.
 4. The apparatus of claim1, further comprising a discriminator network configured to compareestimated training output data with training synthetic output dataduring training, the DNN and the discriminator network corresponding toa generative adversarial network (GAN) during training.
 5. The apparatusof claim 1, wherein the DNN further comprises a second convolutionallayer configured to receive optical property data, the plurality ofintermediate layers comprises a concatenate layer, and the estimated FLIoutput data is further related to the optical property data.
 6. Theapparatus of claim 1, wherein the first convolutional layer is athree-dimensional (3D) convolutional layer, the plurality ofintermediate layers comprises a second 3D convolutional layer, a reshapelayer, a separable two-dimensional (2D) convolutional layer, and a 2Dexception block, and the DNN comprises a plurality of output layers andeach output layer is a coefficient block that corresponds to the fullyconvolutional (FC) down-sample layer.
 7. The apparatus of claim 1,wherein the FLI input data is selected from the group comprising visibleFLI microscopy (FLIM) data, near infrared (NIR) FLIM data and NIR gatedmacroscopy FLI (MFLI) data.
 8. A method for fluorescence lifetimeimaging (FLI), the method comprising: receiving, by a firstconvolutional layer of a deep neural network (DNN), FLI input data;receiving, by each intermediate layer of a plurality of intermediatelayers, a respective intermediate input corresponding to an output of arespective prior layer; providing, by each intermediate layer, arespective intermediate output related to the received respectiveintermediate input; and providing, by an output layer, estimated FLIoutput data corresponding to the received FLI input data, the outputlayer is a fully convolutional (FC) down-sample layer comprising: afirst two-dimensional (2D) convolutional block; a first intermediateblock coupled to an output of the first 2D convolutional block; a second2D convolutional block coupled to an output of the first intermediateblock; a second intermediate block coupled to an output of the second 2Dconvolutional block; a third 2D convolutional block coupled to an outputof the second intermediate block; and a rectified linear unit blockcoupled to an output of the third 2D convolutional block; wherein thefirst intermediate block and the second intermediate block each comprisea batch normalization function and a rectified linear unit wherein theDNN is trained using synthetic data.
 9. The method of claim 8, whereinthe first convolutional layer is a three-dimensional (3D) convolutionallayer, the plurality of intermediate layers comprises a 3D residualblock, a reshape layer, a two-dimensional (2D) convolutional layer, andat least one 2D residual block.
 10. The method of claim 8, wherein thefirst convolutional layer is a separable two-dimensional (2D)convolutional layer; the plurality of intermediate layers comprises a 2Dexception block, a 2D convolutional layer.
 11. The method of claim 8,further comprising comparing, by a discriminator network, estimatedtraining output data with training synthetic output data duringtraining, the DNN and the discriminator network corresponding to agenerative adversarial network (GAN) during training.
 12. The method ofclaim 8, further comprising receiving, by a second convolutional layer,optical property data, the plurality of intermediate layers comprising aconcatenate layer, and the estimated FLI output data is further relatedto the optical property data.
 13. The method of claim 8, wherein thefirst convolutional layer is a three-dimensional (3D) convolutionallayer, the plurality of intermediate layers comprises a second 3Dconvolutional layer, a reshape layer, a separable two-dimensional (2D)convolutional layer, and a 2D exception block, and the DNN comprises aplurality of output layers and each output layer is a coefficient blockthat corresponds to the fully convolutional (FC) down-sample layer. 14.The method of claim 8, wherein the FLI input data is selected from thegroup comprising visible FLI microscopy (FLIM) data, near infrared (NIR)FLIM data and NIR gated macroscopy FLI (MFLI) data.
 15. A fluorescencelifetime imaging (FLI) deep learning system, the system comprising: aprocessor; a memory; input/output circuitry; and a deep neural network(DNN), the DNN comprising: a first convolutional layer configured toreceive FLI input data; a plurality of intermediate layers, eachintermediate layer configured to receive a respective intermediate inputcorresponding to an output of a respective prior layer, eachintermediate layer further configured to provide a respectiveintermediate output related to the received respective intermediateinput; and an output layer configured to provide estimated FLI outputdata corresponding to the received FLI input data, the output layer is afully convolutional (FC) down-sample layer comprising: a firsttwo-dimensional (2D) convolutional block; a first intermediate blockcoupled to an output of the first 2D convolutional block; a second 2Dconvolutional block coupled to an output of the first intermediateblock; a second intermediate block coupled to an output of the second 2Dconvolutional block; a third 2D convolutional block coupled to an outputof the second intermediate block; and a rectified linear unit blockcoupled to an output of the third 2D convolutional block; wherein thefirst intermediate block and the second intermediate block each comprisea batch normalization function and a rectified linear unit; wherein theDNN is trained using synthetic data.
 16. The system of claim 15, whereinthe first convolutional layer is a three-dimensional (3D) convolutionallayer, the plurality of intermediate layers comprises a 3D residualblock, a reshape layer, a two-dimensional (2D) convolutional layer, andat least one 2D residual block.
 17. The system of claim 15, wherein thefirst convolutional layer is a separable two-dimensional (2D)convolutional layer; the plurality of intermediate layers comprises a 2Dexception block, a 2D convolutional layer, and a 2D residual block. 18.The system of claim 15, further comprising a discriminator networkconfigured to compare estimated training output data with trainingsynthetic output data during training, the DNN and the discriminatornetwork corresponding to a generative adversarial network (GAN) duringtraining.
 19. The system of claim 15, wherein the DNN further comprisesa second convolutional layer configured to receive optical propertydata, the plurality of intermediate layers comprises a concatenatelayer, and the estimated FLI output data is further related to theoptical property data.
 20. The system of claim 15, wherein the firstconvolutional layer is a three-dimensional (3D) convolutional layer, theplurality of intermediate layers comprises a second 3D convolutionallayer, a reshape layer, a separable two-dimensional (2D) convolutionallayer, and a 2D exception block, and the DNN comprises a plurality ofoutput layers and each output layer is a coefficient block thatcorresponds to the fully convolutional (FC) down-sample layer.